Monitoring the performance of a content player

ABSTRACT

Improving a content player engagement is described. An engagement of a first content player with respect to a content item being downloaded by the first client is measured. Performance information associated with the first content player is obtained. A quantitative relationship between the engagement and the performance information is determined. How a second client obtains the same content is adjusted based at least in part on the determined quantitative relationship.

CROSS REFERENCE TO OTHER APPLICATIONS

This application is a continuation of co-pending U.S. patent applicationSer. No. 12/780,799, entitled MONITORING THE PERFORMANCE OF A CONTENTPLAYER filed May 14, 2010 which is incorporated herein by reference forall purposes, which claims priority to U.S. Provisional PatentApplication No. 61/227,066 entitled REAL-TIME TELEMETRY FOR CONTENTfiled Jul. 20, 2009 and to U.S. Provisional Patent Application No.61/339,925 entitled TELEMETRY FOR CONTENT filed Mar. 10, 2010, both ofwhich are incorporated herein by reference for all purposes.

BACKGROUND OF THE INVENTION

Users are increasingly using networks such as the Internet to accesscontent, such as video files and live streaming/video on demand content,via client machines. Such content is often large, time sensitive, orboth. As demand for such content increases, there are challenges indistributing that content efficiently and with high quality. As oneexample, it can be difficult to determine what impact modifications tothe delivery of the content have on user experience.

BRIEF DESCRIPTION OF THE DRAWINGS

Various embodiments of the invention are disclosed in the followingdetailed description and the accompanying drawings.

FIG. 1 is an illustration of an environment in which content isdistributed.

FIG. 2A illustrates an embodiment of a client.

FIG. 2B illustrates an embodiment of a client.

FIG. 3 illustrates an example of a process for monitoring theperformance of a content player.

FIG. 4 illustrates an example of a process for dynamically adjusting aheartbeat.

FIG. 5 is an illustration of an environment in which status informationis received and processed.

FIG. 6 illustrates an example of a process for detecting a problem in acontent distribution.

FIG. 7 illustrates an example of a process for correcting a problem in acontent distribution.

FIG. 8 illustrates an example of an environment in which content isdistributed.

FIG. 9 illustrates an embodiment of an interface through which contentdistribution monitoring data is exposed.

FIG. 10 illustrates an embodiment of an interface through which contentdistribution monitoring data is exposed

FIG. 11 illustrates an embodiment of an interface through which contentdistribution monitoring data is exposed

FIG. 12 illustrates an embodiment of an interface through which contentdistribution monitoring data is exposed.

FIG. 13 illustrates an embodiment of an interface through which contentdistribution monitoring data is exposed.

FIG. 14A is a graph illustrating the impact of distribution quality on acontent item.

FIG. 14B is a graph illustrating the impact of distribution quality on acontent item.

FIG. 15 illustrates an example of a process for improving a contentplayer engagement.

DETAILED DESCRIPTION

The invention can be implemented in numerous ways, including as aprocess; an apparatus; a system; a composition of matter; a computerprogram product embodied on a computer readable storage medium; and/or aprocessor, such as a processor configured to execute instructions storedon and/or provided by a memory coupled to the processor. In thisspecification, these implementations, or any other form that theinvention may take, may be referred to as techniques. In general, theorder of the steps of disclosed processes may be altered within thescope of the invention. Unless stated otherwise, a component such as aprocessor or a memory described as being configured to perform a taskmay be implemented as a general component that is temporarily configuredto perform the task at a given time or a specific component that ismanufactured to perform the task. As used herein, the term ‘processor’refers to one or more devices, circuits, and/or processing coresconfigured to process data, such as computer program instructions.

A detailed description of one or more embodiments of the invention isprovided below along with accompanying figures that illustrate theprinciples of the invention. The invention is described in connectionwith such embodiments, but the invention is not limited to anyembodiment. The scope of the invention is limited only by the claims andthe invention encompasses numerous alternatives, modifications andequivalents. Numerous specific details are set forth in the followingdescription in order to provide a thorough understanding of theinvention. These details are provided for the purpose of example and theinvention may be practiced according to the claims without some or allof these specific details. For the purpose of clarity, technicalmaterial that is known in the technical fields related to the inventionhas not been described in detail so that the invention is notunnecessarily obscured.

FIG. 1 is an illustration of an environment in which content isdistributed. In the example shown, clients 170-184 are used to accesscontent, such as audiovisual content (e.g., movies, songs, televisionshows, sporting events, games, images, etc.) that is owned by contentowners. The content is stored (or captured) at origin servers 196-198,then distributed via other servers, caches, content distributionnetworks (CDNs), proxies, etc. (collectively, “content sources”).Content sources employ a variety of technologies and include HTTP, AdobeFlash Media, and Microsoft Internet Information Service servers. In someembodiments content is also distributed by clients (e.g., usingpeer-to-peer techniques).

Examples of clients include personal computers (170), laptops (182),cellular phones/personal digital assistants (178), and other types ofinformation appliances (not shown) such as set-top boxes, game consoles,broadband routers, file servers, video servers, and digital videorecorders, as applicable. The clients shown are used by subscribers tovarious Internet service providers (ISPs). For example, clients 170,172, and 174 are subscribed to SP1 (122), while clients 176, 178, and180 are subscribed to SP2 (124), and clients 182 and 184 are subscribedto SP3 (126).

In the example shown, a movie studio (“Studio”) has contracted withcontent distributor 142 to provide downloadable copies of its films inexchange for a fee. Similarly, a television network (“XYZ”) hascontracted with content distributors 142-146 to provide viewers withaccess to live streams of its broadcasts as well as streams oftelevision show episodes and sporting events. In some cases, the contentdistributor is owned/operated by the content owner.

Content distributor 142 has a data center that is provided with networkaccess by backbone ISP 132. Though represented here by a single node(also referred to herein as a “CDN node”), content distributor 142 maytypically have multiple data centers (not shown) and may make use ofmultiple backbone or other ISPs. Content distributor 144 has a datacenter that is provided with network access by backbone ISP 134.Advertisements are served to various clients via ad servers 150-152.

Suppose a user of client 172 (hereinafter “Alice”) would like to watch alive soccer game owned by XYZ. Client 172 includes a web browserapplication. Alice uses the web browser application to navigate to aportal owned by XYZ, such as “http://xyztvnetwork.com/livegames.” Herrequest for the game is directed to a CDN node that is closest to her.In this case, CDN 146 is the fewest hops away from her client. Herclient then begins streaming the content from CDN 146, which is in turnrendered in her browser (e.g., via a Flash or Silverlight player).Advertisements, associated with the portal, are served to her by adserver 150.

In addition to CDN 146 and ad server 150, Alice's client is also incommunication with content distribution monitor 102. As will bedescribed in more detail below, client 172 provides status information(also referred to herein as a “heartbeat”), on a recurring basis, tocontent distribution monitor 102.

The status information includes a variety of telemetry data such asinformation that captures the quality of the user experience (e.g.,video stream quality), and information pertaining to user behavior.Examples of quality metrics include: the length of time it takes for thesoccer game video to start playing, the number of buffering events (ifany), the length of buffering events, and the number of frames persecond rendered by the video player. Examples of user behavior include:starting and stopping playing a video or audio stream, seeking withinthe stream, switching the player to full screen mode,minimizing/restoring the player, a change in the volume level of theplayer, and clicking on an advertisement.

As other users of clients 170-184 request content, their respectiveplayers similarly obtain content from content sources such as CDN 144and also communicate status information (also referred to herein astelemetry information) to content distribution monitor 102. Such playersmay be browser-based as with Alice's, or they may be standaloneapplications, as applicable. In various embodiments, all clients in theenvironment provide status information to content distribution monitor102.

As will be described in more detail below, different clients may providecontent distribution monitor 102 with different levels of detail, andmay also do so with differing frequency. For example, client 178 is asmartphone with less powerful hardware than client 172 and more limitedbandwidth. It is configured to provide less information to contentdistribution monitor 102 than client 172 and also does so lessfrequently than client 172.

Content distribution monitor 102 collects and processes the informationreceived from Alice's client along with other clients. The collectedinformation is made available in real-time to control entities/operatorsand can be used to detect and remedy problems in the contentdistribution. Examples of such problems include excessive buffering,freezing, and frame skipping. Additional information pertaining todelivery resources (e.g., CDN 142) and network providers (e.g., ISP 126)is also made available, as is other information pertaining to clientssuch as demographic information.

In the example shown in FIG. 1, a single content distribution monitor102 is used. Portions of content distribution monitor 102 may beprovided by and/or replicated across various other modules orinfrastructure depending, for example, on factors such as scalabilityand availability (reducing the likelihood of having a single point offailure), and the techniques described herein may be adaptedaccordingly. In some embodiments content distribution monitor 102 isimplemented across a set of machines distributed among several datacenters. A Resilience Service Layer (RSL) can also be used to ensurethat the monitoring service is not disrupted when/if a subset ofmachines fail or a subset of data centers hosting the contentdistribution monitor are disconnected from the Internet.

Examples of Client Architecture

In various embodiments, the collection of status information and thereporting of that information to the content distribution manager areperformed by a “monitoring module” included in the client.

FIG. 2A illustrates an embodiment of a client. In the example shown,client 202 includes a content player application 204 which in turnincorporates monitoring module 206. Content is received by the playerfrom content source 208. Monitoring module 206 is in communication withcontent distribution monitor 210.

In the example shown, module 206 is implemented in ActionScript anddeployed as an SWF library that is dynamically loaded by player 204. InFlash, the NetStream( ) class is mainly responsible for streaming,buffering, and playing the video. The monitor is implemented as anelement in the content player which wraps the video element, andprovides the same interface as the video element, as described in moredetail below.

Module 206 can also implemented using other approaches, such as in the.NET platform for Silverlight and deployed as a DLL library that isdynamically loaded by player application 204. In Silverlight, theMediaElement( ) class is mainly responsible for streaming, buffering,and playing the video. The monitor is implemented as an element in thecontent player which wraps the video element, and provides the sameinterface as the video element, as described in more detail below.

Examples of some of the information collected by monitoring module 206include the following, expressed in key-value pairs:

-   -   (player_state, “buffering”): The stream is buffering.    -   (buffer_length, 5 s): The current buffer length is five seconds.    -   (join_time, 3 s): The join time was 3 sec.    -   (frame_per_second, 30): The number of frames per second is 30.    -   (player-mode, “Full Screen”): The player is running in        full-screen mode.

FIG. 2B illustrates an embodiment of a client. In the example shown,client 252 includes a web browser application which in turn incorporatesmonitoring module 256. Content is received by the player from contentsource 258. Monitoring module 256 is in communication with contentdistribution monitor 260. In the example shown, module 256 isimplemented in JavaScript. The monitor periodically collects informationabout the current web page and the browser. Examples of some of theinformation collected by monitoring module 256 include the following,expressed in key, value pairs:

-   -   (browser_minimized, “yes”): The browser window is minimized.    -   (tab_visible, “no”): The tab containing the web page is not        visible.    -   (pointer_pos, “x, y”): The position of the pointer on the web        page.    -   (banner_display”, “ACMECars”): The banner ad on the web page is        for ACME Cars.

As explained above, the monitoring module can be implemented in avariety of ways. For example, the monitoring module can be included inthe client's player by the author of the player. The monitoring modulecan also extend the functionality of an existing player, such as bybeing implemented as a plugin, library, or standalone application (e.g.,a helper application). Various examples of techniques for integrating amonitoring module with an existing player will now be described.

In one embodiment, the monitoring module is a wrapper around the lowestlevel streamer module (e.g., “NetStream” in Flash, or “MediaElement” inSilverlight). The player uses the wrapper, which provides the same orenhanced version of the API provided by the streamer module, to streamvideo. Logic in the wrapper captures the appropriate telemetry data.

Example. “ConvivaNetStream” extends and wraps “NetStream”:

-   -   var ns:NetStream=new ConvivaNetStream( )    -   ns.play(<stream>);

In a second embodiment, the monitoring module exists as a sideattachment and is not present in the code path for streaming to occur.The player passes the streamer module to the monitoring module eitherdirectly or through a proxy module. The monitoring module readsproperties and listens for events from the streamer module or proxymodule to collect data. The proxy module method prevents the monitoringmodule from interfering with the streaming.

Example 1.

-   -   var ns:NetStream=new NetStream( )    -   ns.play(<stream>);    -   LivePass.createMonitoringSession(ns);

Example 2. “NetStreamProxy” wraps “NetStream,” but prohibits any callsthat may adversely impact “NetStream”:

-   -   var ns:NetStream=new NetStream( )    -   var nsProxy:NetStreamProxy=new NetStreamProxy(ns);    -   ns.play(<stream>);    -   LivePass.createMonitoringSession(nsProxy);

In a third embodiment, the monitoring module is built into the streamermodule. The player uses the streamer module. The monitoring modulewithin collects data.

Example

-   -   var ns:NetStream=new NetStream( )    -   ns.play(<stream>);

Additional Status Information Examples

As mentioned above, clients downloading/streaming content from contentproviders are also in communication with content distribution monitor102 on a recurring, time-driven basis. Each heartbeat contains asnapshot of the session state at the time the heartbeat was sent out anda summary of the events and measurements since the last heartbeat orsince the start of the session, as applicable. Examples of informationthat can be included in a heartbeat (though need not be present in everyheartbeat) include the following:

-   -   version: The version of the heartbeat.    -   clientID: A unique identifier associated with the client's        monitoring module (described in more detail below) when it is        downloaded/installed on the client.    -   clientVersion: The version of the monitoring module.    -   customerID: An identifier associated with an entity such as XYZ        or Studio.    -   sessionID: A unique identifier associated with the content        viewing session.    -   objectID: A unique identifier associated with the content being        streamed. In Alice's case, this is the soccer game.    -   currentResource: The source from which the content is currently        being obtained. In Alice's case, this is CDN 146.    -   candiadateResourceList: A list of potential sources from which        the player can obtain the content. In Alice's case, this could        include CDN 144, CDN 142, cache 148, etc. The “currentResource”        is one resource in the “candidateResourceList.”    -   resourceUsage: The number of bytes loaded/streamed since the        start of the session.    -   currentBufferSize: The current buffer size.    -   minBufferSize: The minimum buffer size during the previous        heartbeat interval.    -   maxBufferSize: The maximum buffer size during the previous        heartbeat interval.    -   numBufferEmptyEvents: The number of buffering empty events since        previous heartbeat.    -   currentBitrate: The bitrate at which the content is currently        streamed.    -   playheadTime: For video-on-demand content (i.e., not a live        event), this is the time offset of the stream file.    -   currentlyPlaying: For live content, this is the time offset of        the stream since the player started.    -   joinTime: The amount of time that it took the player to enter        the player state for the session.    -   averageFPS: The average rendering frame per second (FPS) at        which the content was rendered since the last heartbeat.    -   encodedFPS: The FPS at which the content was encoded.    -   averageFPS: Can be lower than the “encodedFPS” if the client's        CPU is unable to render the content at the “encodedFPS” or if        the client cannot stream the content fast enough.    -   totalPlayTime: The total play time since the beginning of the        session.    -   totalBufferingTime: The total buffering time since the beginning        of the session.    -   totalPauseTime: The total pause time since the beginning of the        session.    -   totalSleepTime: The total sleep time since the beginning of the        session. The client is in a sleep state if the client is        suspended.    -   sessionTime: The total time elapsed since the session started.    -   currentState: The state of the player when the heartbeat was        sent. Examples of player states include: play, pause, buffering,        joining, and seeking.    -   numRateSwitches: The number of bitrate switches since the        beginning of the session.    -   numResourceSwitches: The number of resource switches since the        beginning of the session.    -   rateSwitchingEvent.time: The time at which a bitrate switch was        attempted, measured from the start of the session.    -   rateSwitchingEvent.from: The bitrate of the stream at the time        the switch was attempted.    -   rateSwitchingEvent.to: The target bitrate of the switch.    -   rateSwitchingEvent.result: Indicates whether the switch was        successful or not. For example, a switch may not be succeed if        the client cannot sustain the “rateSwitchingEvent.result”        bitrate.    -   resourceSwitchingEvent.time: The time at which a resource switch        was attempted, measured from the start of the session.    -   resourceSwitchingEvent.from: The resource from which the content        was streamed at the time the switch was attempted.    -   resourceSwitchingEvent.to: The target resource for the switch.    -   resourceSwitchingEvent.results: Indicates whether the switch was        successful or not.    -   errorList: A list of errors encountered from the start of the        session.

FIG. 3 illustrates an example of a process for monitoring theperformance of a content player. In some embodiments the process shownin FIG. 3 is performed by a client such as client 172. The processbegins at 302 when information relating to the state of a content playeris obtained. For example, at 302, client 172 collects variousinformation pertaining to the video player that Alice uses to watch thestreaming soccer game. Information such as the amount of time anadvertisement took to load and which ad server supplied theadvertisement is also obtained. In some embodiments, additionalprocessing is performed in the collected information. Examples includecomputing averages, minimums, maximums, and the x^(th) percentile overthe last few samples.

At 304, at least a portion of the obtained information is reported to acontent distribution monitoring server. For example, at 304, client 172sends a heartbeat to content distribution monitor 102, including some orall of the information collected at 302. Information sent at 304 canalso include processed results such averages and minimums, instead of orin addition to raw data.

In various embodiments, the process shown in FIG. 3 repeats throughout acontent playing session. The process may repeat regularly (e.g., onceevery second), and may also repeat with varying frequency.

The amount of data reported by the client in a heartbeat to contentdistribution monitor 102 can affect the scalability of the contentdistribution monitor. The amount of data collected and reported by theclient can also potentially impact the performance of the client. If theclient is otherwise resource constrained (e.g., due to other processes,or due to hardware limitations), adjustments can be made to how muchinformation is collected and with what frequency. For example, supposethe buffer size is expected to be of size three seconds. The time periodemployed by the monitoring module can be set to one second or a fewhundred milliseconds. The adjustment can be dynamically adjusted asneeded. The period can be decreased if the buffer grows and increased ifthe buffer shrinks, thus minimizing overhead while still being able todetect a low buffer size which can impact the video quality.

The amount of data sent can be adjusted according to two parameters: theheartbeat “period,” and the size of heartbeats, both of which can beadaptively changed by either the client or the content distributionmonitor as needed. For example, if the quality experienced by the clientis acceptable, the client may reduce the heartbeat frequency. This canentail collecting state information less frequently and can also entailsending collected information less frequently. If the quality degrades,the client can increase the heartbeat frequency accordingly. As oneexample, client 172 can employ a rule that, as long as the bufferingratio is less than or equal to 0.5%, the heartbeat period is 30 seconds.If the buffering ratio is between 0.5% and 1% the heartbeat period isadjusted to 20 seconds. If the buffering ratio is equal to or greaterthan 1%, the heartbeat period is adjusted to 10 seconds.

Content distribution monitor 102 can also direct client 172 to increaseor decrease the heartbeat frequency based on factors such as the currentload on the content distribution monitor (or applicable componentthereof). For example, if the load on content distribution monitor 102exceeds a predefined threshold, clients are instructed to increase theheartbeat interval and/or reduce the detail of the information sent in aheartbeat.

As one example, content distribution monitor 102 can employ a rule that,if its resource utilization exceeds 80%, clients are instructed todouble their respective heartbeat intervals. Monitor 102 can alsoselectively send such instructions to clients. For example, clients withperiods under 30 seconds can be instructed to double their periods,while clients with periods above 30 seconds are not sent suchinstructions.

As another example, when monitor 102's utilization exceeds 80%, it caninstruct clients to reduce heartbeat size by sending less information.This can be less detailed information (e.g., sending only the number ofresource switches, instead of the detailed information about theseswitches), or can include aggregate measurements about certain eventsinstead of sending individual details. If the client is alreadyaggregating certain information, the measurements can be aggregated overa longer time interval (e.g., 10 seconds instead of 2 seconds).Additional examples follow.

-   -   Instead of sending every buffering event, send the aggregate        time the session spent in the buffering state since the last        heartbeat.    -   Instead of sending every measurement of the rendering rate, send        the average over the entire heartbeat interval.    -   Instead of sending information about a switch event at the time        when the switch event happened, store the event on the client        and send it with the next heartbeat.

FIG. 4 illustrates an example of a process for dynamically adjusting aheartbeat. In some embodiments the process shown in FIG. 4 is performedby a client such as client 172. The process begins at 402 when anindication is received that a change should be made to either thecollection, or reporting of status information (or both). As explainedabove, the indication can be received as an instruction from contentdistribution monitoring server 102, and can also be received as theresult of a determination made by the client itself. At 404, anadjustment is made to the collection/reporting of status informationimplicated by the indication received at 402.

Inferring State Information

In some cases, the monitoring and reporting functionality describedherein as being provided by a client is wholly integrated with theplayer itself. For example, Studio might make available a custom playerapplication that can be installed on a client by users that want towatch Studio films. The customer player includes all of the necessarylogic to provide complete heartbeat information, and also includes logicfor communicating with the content distribution monitor, adjusting thefrequency with which status information is communicated to the contentdistribution monitor, and other functionality described herein. However,depending on the player deployed on a client, not all of the exampleheartbeat information described above may be directly accessible forcollection and transmission. For example, the Flash player plugin usedby Alice to view the soccer game does not provide direct access to itsstate. Using the techniques described herein, content distributionmonitor 102 can nonetheless be provided with the state of the player.

In some embodiments, a monitoring module is embedded in the contentplayer deployed on the client. For example, when Alice directs herbrowser to XYZ's website, a custom Flash-based player is loaded. Thecustom player has direct access to the API of the video plugin. Possiblestates of the player are as follows:

-   -   Playing: The player's buffer is not empty, and the player        renders frames on the screen.    -   Buffering: The player's buffer is either empty or is less than a        predefined threshold, making it impossible for the player to        render new frames.    -   Joining: The player has just connected and has not yet        accumulated enough data in its buffer to start rendering frames.    -   Pause: The player is paused, usually, as a result of a user        action. During the pause state, no new frames are rendered even        if the buffer is full.    -   Sleep: The client is suspended.

In the case of the aforementioned Flash player, NetStatus eventsprovided by the player can be used to infer the state, as describedbelow:

-   -   Buffering: When one of a NetStatus.Buffer.Empty,        NetStream.Play.Start, or NetStream.Play.Reset events is        received, an inference is made that the player is buffering data        and the video is not playing. For live RTMP streams, the        mentioned events might not be available. In that case, if the        playhead time has stopped moving and there is no data in the        content player buffer, an inference is made that the player is        buffering data and the video is not playing.    -   Playing: When a NetStatus.Buffer.Full event is received, an        inference is made that the player is playing video. If the        player is currently in a Paused state and the playheadTime        starts to move, an inference is made that the player is playing        video.    -   Paused: If the playheadTime does not progress for 1.6 seconds,        an inference is made that the player is paused.    -   Sleep: If the amount of time that has elapsed between two        firings of a periodic one second timer is greater than 30        seconds, an inference is made that the client has been in a        sleep mode since the last firing of the timer.    -   Stopped: For HTTP streams, when a NetStatus.Play.Stop event is        received, an inference is made that the stream has ended. For        RTMP streams, when a NetStatus.Play.Complete event is received,        an inference is made that the stream has ended. When a        NetStatus.Play.Stop is received, an inference is made that the        download has finished, but the stream is still playing.    -   Error: If the stream stays in stopped state for fifteen seconds        at the start of a session, an inference is made that it has        failed to connect.

In addition to the player states described above, an additional statecan also be inferred:

-   -   Zombie state: An inference is made that the player is in a        zombie state (a non-operating state with no user actively using        it) if it persists in a non-playing state for more than a        specified time. Once the player is in a zombie state, the        transmission of heartbeats to content distribution monitor 102        is halted. Reporting will resume if the player returns to a        playing state. Non-playing states include: “Buffering,”        “Paused,” “Stopped,” and “Error.”

In addition to the player states, the monitoring module monitors variousmetrics, including the number of bytes downloaded/streamed since thesession started, the bitrate of the stream, and the rendering rate.

Some players, such as the Flash 9 plugin, do not provide a direct APIfrom which the number of bytes streamed/downloaded or the streambitrate. Nonetheless, a variety of techniques can be used to obtain orestimate the bitrate. As one example, the developer of the player mightexpose the bitrate through an API call.

Another technique for obtaining a bitrate is to use metadata associatedwith the content. For example, suppose metadata is made availablethrough a configuration file associated to the content. Theconfiguration resides on an origin server such as origin server 198 andalso includes other information such as the title of the content, itsgenre, and a list of CDNs where the content is available.

Yet another technique for obtaining a bitrate is to examine the contentURL. For example, a movie might be accessible at the URLhttp://www.CDN-C.com/Studio/JaneEyre300 Kbps. In this case, the bitrateof the content is likely to be 300 Kbps.

An estimation of the stream bitrate can also be made. The estimation,along with the amount of time the player was in the playing state, andthe size of the buffer, are used to estimate the number of bytesdownloaded/streamed. As one example, suppose a player streamed data at400 Kbps, was in a playing state for 315 seconds, and at the time oftaking the measurement the buffer size contained data for playinganother 15 seconds. The total number of bytes downloaded is: 400Kbps*(315 seconds+15 seconds)=16 MB.

In the event that the bitrate is available through multiple of the abovetechniques, in various embodiments each applicable approach is used, ina specified order, until a valid bitrate (e.g., greater than 0 and lessthan 10,000 kbps) is obtained.

The number of bytes downloaded can be estimated as(totalPlayingTime+bufferLength)*bitrate. If multiple bitrates are used,in some embodiments the bytes downloaded is estimated as the sum oftotalPlayingTime[bitrate] *bit-rate+bufferLength*currentBitrate.

Rendering quality is another example metric that can be obtained andprovided to content distribution coordinator 102. Rendering quality isthe ratio between the frames per second (FPS) rendered by the player andthe FPS at which the stream was encoded.

Some players do not directly provide the information needed to computethe rendering quality. For example, Flash 9 has an API that exposesrendered frames per second, but not encoded frames per second. Onetechnique to compute the encoded FPS, in such a scenario, is as follows.The rendering FPS is measured and an assumption is made that the encodedFPS is the maximum FPS observed over the course of the session. Anothertechnique is to estimate the encoded FPS as the maximum of a threesecond sliding average of rendered FPS sampled five times a second.

In order to ensure accurate accounting of the player status over thecourse of the session, it is important to determine the time intervalswhen the system is in sleep or suspend mode. These intervals can bedetected using notification APIs provided by the system for thispurpose. For systems that do not provide such API (e.g., Flash Player,or Silverlight), the intervals can be estimated by setting a timer tofire at periodic intervals (e.g, every 1 second). At each firing of thetimer the current system time is recorded, and the time elapsed sincethe last firing is computed. If the time elapsed is greater that a giventhreshold (e.g., 10 seconds), an inference is made that the givenelapsed time interval was spent in sleep or suspend mode.

Inferring Player Capability Information

The following are example player capabilities that impact the quality ofthe video experience: processing speed, video rendering capabilities,amount of memory available, and amount of bandwidth available.

-   -   Processing speed: The monitoring module can use the API of the        content player, or the underlying platform to read the        processing speed of the player. For example, if the content        player is implemented on the Silverlight platform the monitor        can use the Environment.ProcessorCount to obtain the count of        the processors available.

For platforms whose API does not provide direct reading of processingspeed capabilities, the monitoring module can derive it by using a timerto measure the time required to perform a fixed CPU-intensivecomputation.

-   -   Video rendering capabilities: Some content players have the        ability to use hardware acceleration to render video. The        monitoring module can determine the speed at which the content        player can render video, either by using the API of the content        player or the underlying platform. For example, a monitoring        module using the Flash platform can use the        flash.system.Capabilities.ScreenResolutionX to determine the        screen resolution.    -   Available memory: The amount of available memory has a direct        influence on how well the content player can support rendering        video while performing the computation required for monitoring        and other user-interaction tasks. The monitoring module can        obtain the amount of available memory by using the API of the        underlying platform. For example, a monitoring module using the        Flash platform can use the API flash.system.System.totalMemory        to termine how much memory is available.    -   Available download bandwidth: The available download bandwidth        has a direct impact on the quality of the video stream that can        be downloaded and played. The monitoring module can infer the        available bandwidth by measuring the time interval it takes to        download a fixed size file from the server. Alternatively, the        underlying platform can provide API that can be used to        determine the download bandwidth. For example, a player using        the Flash platform can use the API        flash.net.NetStreamInfo.currentBytesPerSecond.

FIG. 5 is an illustration of an environment in which status informationis received and processed. In various embodiments, the services providedby content distribution monitor 102 are implemented across a scalableinfrastructure, particularly in embodiments where telemetry data isreceived from all clients. In the example shown, the elements containedwithin dashed region 502 collectively provide the functionality ofcontent distribution monitor 102. Each of the layers (e.g., dispatcherlayer 520) is horizontally scalable and their respective components canbe implemented on standard commercially available server hardware (e.g.,having a multi-core processor, 4G+ of RAM, and Gigabit network interfaceadaptors) running a typical server-class operating system (e.g., Linux).

Clients 504-512 each include a monitoring module that collects statusinformation. When the monitoring module on a client is activated, theclient is mapped to a dispatcher server. As one example, when themonitoring module starts, it reads a configuration file that includes alist of dispatcher servers. The monitoring module selects a dispatcherserver at random from the list.

A dispatcher server (514) includes two conceptual modules. The firstmodule implements a communication interface for receiving statusinformation from clients. In some embodiments the module is implementedusing an off-the-shelf web server, and allows clients to connect overthe HTTP protocol (and also allows clients to securely communicate viaSSL). Data received by the first module is passed to the second module.The second module normalizes the data (to a format suitable for furtherprocessing) and passes the normalized data to a real-time streamprocessing component (516).

The real-time stream processing (RSP) layer includes an optimizedsoftware component that processes the telemetry data that it receivesfrom the dispatcher in real-time. A dispatcher sends all heartbeatsbelonging to the same session to the same RSP component.

In some embodiments the RSP component is implemented as a continuouslyrunning service that reads and processes the telemetry data receivedfrom dispatchers via the network over TCP. The telemetry data streamcomprises individual records, each of which represents the telemetrydata sent by the monitoring module. The RSP component reads network dataone record at a time and parses the data into a local datarepresentation. The data received by the RSP component can be stored asin-memory hash tables of records allowing fast execution, and very highthroughputs. Since the RSP component does not require old information,it can periodically purge the in-memory hash tables and increasescalability accordingly. In other embodiments, optimized in-memorydatabases are used.

A mapping function to map heartbeats having a session identifier “ID” toa particular RSP component “i” as follows:

i=hash(ID) mod m,

where hash( ) is a hash function and “m” is the total number of RSPcomponents.

Once an RSP component parses the data records, it performs two maintasks. First, it performs data filtering. A filter is a logicalexpression and is installed at each RSP component instance. As oneexample, the following filter would identify viewers located in SanFrancisco, connected to ISP SP1, streaming from CDN A, one of twoparticular shows:

(city=“San Francisco” AND ISP=“SP1” AND CDN=“CDN A” AND((show=“NewsAt10”) OR (show=“SundayMagazine”))

For each message of incoming telemetry data, the (key, value) pairs inthe record are matched against the filter. If the filter is matched, thedata is associated with the filter.

The second task performed is to compute snapshots and on-line statisticsover the telemetry data matching each filter. One example of a snapshotis the number of players that are in a particular state (e.g.,“playing”). The RSP component generates sequences of these snapshots(e.g., one every second). Examples of statistics computed by the RSPcomponent include: the average number of bytes played over all videostreams matching a filter over a given time interval (e.g., 10 seconds)and the minimum frames per second experienced by a stream matching afilter over a time interval. Snapshots and statistics are updatedcontinuously, from new telemetry data received from clients.

The RSP component provides its computed snapshots and statistics to areal-time global aggregation component (518). The real-time globalaggregation (RTGA) component aggregates the information provided by theRSP component for each filter specified by a user (described in moredetail below).

As explained above, each RSP component (516) receives (via a dispatcher)telemetry data from a subset of the monitoring modules and calculatessnapshots and statistics for all filters. Each RGA component instance isin turn responsible for a subset of the filters. Based on the identifierof the filter, all RSP components send data for that filter to a singleRGA component. The RGA component combines the data from all RSPcomponents for the filters that it is responsible for, resulting in aglobal snapshot and statistics based on information from all monitoringmodules. Examples of aggregation operations performed by an RGAcomponent include: counting the total number of viewers that match aparticular filter, determining the percentage of viewers matching agiven filter that are in buffering state, the join time distributionexperienced by viewers joining a given stream, the current number ofviewers in a given city, the rank of the most popular live events, andso on.

In some embodiments an RGA component's functionality is implemented as acontinuously running service. It reads records sent by the RSPsasynchronously, thus achieving a high throughput. The RGA componentstores the records it receives in in-memory hash tables allowingoptimized access for real-time processing. Old information isperiodically purged from the hash tables to improve efficiency.

As shown in FIG. 5, gateway 522 provides a web service API 524 foraccessing data. Through the API, RGAs data is available on a per-filterbasis. In addition, it also exposes APIs to edit and install new filtersand aggregation functions. In some embodiments gateway 522 isimplemented using an off-the shelf web server (such as Apache) withcustomized code to handle the various web service API calls. Thehandlers return data for a given web API call in various data formatsincluding XML, JSON, SOAP, and HTML, as applicable. The handlers serveas middleware for querying and interactively controlling the RSPs andRGAs.

Gateway 522 also provides access controls and persistently stores theinformation regarding the API requests, data access and presentationpolicies, and filter descriptions. The information is maintained in apersistent database, such as mySQL or Oracle database.

Automatically Detecting and Resolving Content Distribution Problems

As explained above, content distribution monitor 102 aggregatestelemetry information from all clients, processes the information, and,as will be explained in more detail below, allows users (e.g., via auser interface) to view the multi-dimensional results in realtime.Examples of dimensions include:

-   -   Client properties: Including browser type and version, player        type and version, operating system, CPU speed, connection type,        IP address, geographic location, language, autonomous system,        and ISP.    -   Content properties: Including category, title, show, episode        number, duration, encoding format, encoding quality, and        language.    -   Content source properties: Including CDN, data center, and        clients served.    -   User properties: Including whether the user (of the client) is a        premium or free user, and returning or first-time visitor.

Accordingly, using the techniques herein, one is able to track, inrealtime, granular information such as the quality experienced byviewers located in Denver, connected to SP2, and streaming videos fromCDN B, using Flash Player 10. Further, by aggregating and correlatingthe data it receives from all clients, content distribution monitor 102exposes, in realtime, the performance of content sources and networkproviders. (ISPs). Problems in content delivery can be automaticallydetected by examining the results.

FIG. 6 illustrates an example of a process for detecting a problem in acontent distribution. In some embodiments the process shown in FIG. 6 isperformed by content distribution monitor 102. The process begins wheninformation associated with a first content player and a second contentplayer is received from a first client (602) and second client (604),respectively. For example, at 602, telemetry information is receivedfrom client 504 by dispatcher 514. At 604, telemetry information isreceived from client 506 at dispatcher 526. At 606, the receivedinformation as aggregated. For example, at 606, distributed processingof the received data is performed by the RSP and RGA layers. Finally, at608, a determination is made from the aggregate information that acontent distribution problem is indicated. For example, at 608, gateway522 determines that the CDN from which client 504 is obtaining contentis experiencing a problem. Specific examples of analysis and diagnosiswill now be given.

Example: Diagnosing Client Problems (Single Client, Single CDN)

Suppose a content owner, such as Studio-A, distributes its content via asingle CDN, such as CDN C. Suppose that the content is encoded atmultiple bitrates B1 and B2 where B1<B2. If a client A is able tosustain the download rate, but detects problems rendering the frames, aninference can be made that the client A likely has problems with its CPUutilization. An example of performing such an inference follows.

Observe the client buffer size over a period of time T (e.g., T=10seconds) when the client is streaming at bitrate B2. Observe therendering quality measured using dropped frames reported by the player.If the client buffer size is greater than B_threshold (e.g.,B_threshold=0.5*B_max) at all times during the period of observation,and the rendering quality is less than R_threshold_1 (say,R_threshold_1=0.6), then a conclusion can be made that the client cannotsustain displaying bit-rate B2 due to CPU issues on the client.

Perform the same observation with the client playing at bit-rate B1. Ifthe client buffer size is greater than B_threshold (e.g.,B_threshold=0.5*B_max) at all times during the period of observation,and the rendering quality is greater than R_threshold_2 (e.g.,R_threshold_2=0.75), then a conclusion can be made that the client cansustain displaying bit-rate B1.

A numerical example is now provided. Consider a setting where a clientstreams a 1 Mbps stream for 10 seconds with buffer size varies between 9and 10 seconds. Suppose that B_max is 10 seconds and B_threshold is 5seconds. Assume that the rendering quality at 1 Mbps is 0.55 over thisinterval. When the client plays a 500 kbps stream instead, the buffersize is in the same range (9-10 seconds), but the rendering quality is0.9 instead. Then, assuming that R_threshold_1=0.6 andR_threshold_1=0.75, a conclusion can be made that the client can sustain500 kbps but not 1 Mbps.

Example: Diagnosing CDN Problems (Single Client, Two CDNs)

Suppose a content owner, such as Studio-A, distributes its content viatwo CDNs, such as CDN D and CDN E. If a client A is able to sustain thedownload rate from CDN D, but not CDN E, then the client can concludethat CDN E has problems streaming to client A. An example of performingsuch an inference follows.

Compute an aggregate quality metric Q using various measurements fromthe client over a time T (e.g., T=60 seconds). Examples of measurementswhich can be directly used as the metric Q across a group of clientsover a time T include:

-   -   buffering ratio: The total number of seconds the client        experienced buffering divided by the total playing time of the        clients during interval of time T.    -   join time: The average joining time over all clients during time        T over a group of clients.    -   join failures: The fraction of cases (across all attempts) where        the join failed during time T.

If the quality Q is greater than a threshold T1 (e.g., T1=0.95) for CDND, but lower than another threshold T2 (e.g., T2=0.50) for CDN E, then aconclusion can be made that CDN E has problems streaming to client A.

One example remedial action that can be taken at the client is to selectCDN D for streaming the current stream, or use it first for futurestreams.

A numerical example is provided. Consider a setting where a client Mstreams from CDN D and experiences a buffering ratio of 0.5 over K1attempts and time T1. Also, client M streams from CDN E, and over K2attempts and time T2 experiences a buffering ratio of 0.02. In thiscase, given that K1 and K2 are above a threshold (e.g., both greaterthan 2), and T1 and T2 are above a threshold (e.g., both are greaterthan 10 minutes), a conclusion can be made that CDN D has problemsstreaming to the client. A further refinement can be performed bycomputing the buffering ratio for each of the K1 (or K2) streamingattempts from the CDNs to the client. In this case, it can be statedthat CDN D has problems streaming to client M if more than 75% of theattempts from CDN D (out of the K1 attempts) have buffering ratiogreater than 0.5, and more than 75% of the attempts from CDN D havebuffering ratio less than 0.02.

Example: Two Clients in the Same Location Using Different ISPs BothStreaming from a Single CDN

Suppose a content owner, such as Studio XYZ, distributes its content viaa single CDN, such as CDN C. The CDN comprises several geographicallydistributed servers. As clients request content, they are assigned to aparticular server based on their geographical location. If many clientsat the same geographical location, but using different ISPs, experiencequality problems, an inference can be made that the CDN servicing thatgeographical location is having a problem. Note that since XYZ also usesthe services of CDN C, the experiences of clients obtaining XYZ'scontent from the same location can be included in the analysis. One waythis analysis can be implemented on content distribution monitor 102 isas follows.

First, obtain location and network information for all clients. Examplesof location information include city, state, country, DMZ code, andgeographic coordinates. The location information can be obtained in avariety of ways, such as by providing the client's IP address to ageo-location service such as Quova. Also obtain network information suchas Autonomous System Number (ASN) information and ISP information forthe client.

Second, group clients based on their ASN and one geographic attribute,such as DMZ code, city, or state. Let G(ASN1, Geo1) denote the set ofclients connected to a specific ASN, ASN1, which have the samegeographic attribute value, Geo1.

Third, compute an aggregate quality Q for each group G(ASN, Geo) overlast T seconds. Denote this aggregate quality by Q(G(ASN, Geo). Examplesof quality metrics are provided above.

Finally, for each Geo attribute value Geo1, check whether there are atleast two group of clients connected via different ASNs. If theaggregate quality of a certain fraction F of the groups is greater thana threshold BufferingThreshold1, then conclude that the CDN isexperiencing problems serving the clients at location Geo1. In someembodiments a minimum group size is defined so that only groups having anumber of clients larger than a given threshold are considered. As oneexample, the fraction F is selected between 0.75 and 1.

A numerical example is now provided. Consider a system setting whereBufferingThreshold1=0.1, MinGroupSize=200, and F=1.0. Based on ananalysis of Their IP addresses, 20,000 viewers are determined to belocated in San Francisco, connected to CDN C, and watching a new releaseby Studio. Assume 5,000 clients are connected to ASN1 belonging to SP1;5,000 clients are connected to ASN2 belonging to SP2; 9,900 clients areconnected to ASN3 belonging to SP3; and 100 clients are connected toASN4 which belongs to a wireless company. Four groups exist: G(SF,ASN1), G(SF, ASN2), G(SF, ASN4), and G(SF, ASN4), respectively.

Each client reports for each session (a) the total time it spent inbuffering during every 10 second interval; and (b) the total playingtime during the same 10 second interval. For example, suppose a userCharlie has the IP address 1.2.3.4 which belong to ASN1, and watches thestream with the URL rtmp://www.CDNC.example.com/12346/video.flv. Charliereports in a particular heartbeat that he has experienced 1 second ofbuffering and 9 seconds of playing time. Another user, Bob, has the IPaddress 2.3.4.5 which belongs to ASN3, and watches the stream with thesame URL. Bob reports in a particular heartbeat that he has experienced0.5 seconds of buffering and 5 seconds of playing time. Bob's playerspent the remaining 5 seconds in pause state. The pause time is not usedin the computation of the buffering ratio.

Content distribution monitor 102 computes over each interval of time T=5minutes, the aggregate average buffering ratio, defined as the (1) thetotal buffering time experienced by all sessions in a group during Tdivided by (2) the total playing time over all sessions in the groupduring the same time interval T. The aggregate quality ratio for each ofthe four groups is as follows:

Q(G(SF, ASN1))=0.14, Q(G(SF, ASN2))=0.21, Q(G(SF, ASN3))=0.18, Q(G(SF,ASN4))=0.04.

Since the number of clients in G(SF, ASN4) is less than MinGroupSize,the diagnosis analysis ignores this group. Also, since the bufferingratio of all remaining groups is greater than BufferingThreshold1, aconclusion is made that CDN C experiences quality issues serving clientsin San Francisco.

Suppose the above example is modified such that ASN4 has 1000 clients,and Q(G(SF, ASN1))=0.02, Q(G(SF, ASN2))=0.21, Q(G(SF, ASN3))=0.18,Q(G(SF, ASN4))=0.04. In this case, an inference can be made that CDN Cdoes not experience quality issues serving clients in San Francisco ingeneral. The problem can then be narrowed down further and smaller ordifferent sets of clients investigated, such as clients that areconnected to ASN1 served by CDN C.

Example: Inferring Problems in a Particular ASN (Single CDN)

Suppose a content owner, such as Studio XYZ, distributes its content viaa single CDN, such as CDN C. If the quality of clients at the samelocation but using different ISPs is very different, an inference can bemade that the quality problems experienced by the clients are due to theISP (and not due to the CDN). One way this analysis can be implementedon content distribution monitor 102 is as follows.

Perform the first three steps of the previous diagnosis (describing thediagnosis of CDN quality issues). As a final step, let G(ASN1, Geo1) bea group experiencing low quality. If there is at least another group,G(ASN2, Geo1) at the same location experiencing high quality, thenconclude that ASN1 is the cause of quality issues. In particular, aconclusion that ASN2 has quality issues is reached if, and only if,Q(G(ASN2, Geo1))−Q(G(ASN1, Geo1))>BufferingRatioDiscrepancy.

A numerical example is now provided. Consider a system setting whereBufferingRatioDiscrepancy=0.1. Suppose 5,000 viewers are determined tobe located in New York, connected to ASN1, and streaming a baseball gamefrom CDN C. Another 4,000 viewers in New York are connected to ASN2 andstreaming the same game from CDN C.

Each client reports for each session (a) the total time it spent inbuffering during every 10 second interval; and (b) the total playingtime during the same 10 second interval. The aggregate quality ratio foreach group as computed by content distribution monitor 102 is: Q(G(NY,ASN1))=0.14, and Q(G(NY, ASN2))=0.03, respectively.

Since Q(G(SF, ASN1))−Q(G(SF, ASN2))>BufferRatioDiscrepancy, the qualityissues are pinpointed as being ASN1.

Example: Inferring Problems with a Particular CDN (Multiple CDNs)

Suppose a content owner, such as XYZ, distributes its content viamultiple CDNs (e.g., CDN A, CDN B, and CDN C). If clients at the samelocation and using the same ISP experience significantly differentquality when connected to different CDNs, the difference in quality canbe attributed to CDNs. One way this analysis can be implemented oncontent distribution monitor 102 is as follows.

First, obtain location and network information for all clients. Second,classify each client based on its ASN, the CDN it gets data from, andone of its geographic attributes (e.g., DMZ code, city, or state). LetG(ASN_(i), CDN1, Geo_(i)) denote the set of clients connected to aspecific ASN and a geographic region (e.g., for i=1, 2, 3, 4, 5),getting data from same CDN, CDN1. One way of determining the particularCDN from which a client is receiving content is to extract it from theURL used by the client to obtain the content.

Third, compute an aggregate quality Q for each group G(ASN, CDN, Geo)over the last T seconds. Denote this aggregate quality by Q(G(ASN, CDN,Geo)).

Finally, check whether there are at least K pairs of groups that sharethe same ASN and geo-location, but get their data from different CDNs,and which experience different quality. In particular, let G(ASN1, CDNi,Geo1) and G(ASN1, CDNj, Geo1) be one such pair of groups. Then ifQ(G(ASNi, CDN1, Geoi))−Q(G(ASNi, CDN2, Geoi))>QualityThreshold2 forgreater than a fraction F of the K pairs, a conclusion is made that CDN2has problems serving clients in general across a large set of regions.Alternatively, if this happens for smaller than a fraction F but anon-zero set, then a conclusion can be made that CDN2 has problemsserving those (ASN, Geo) combinations for which the difference inquality exceeds the quality threshold.

In some embodiments, a minimum group size is defined so that only groupshaving a number of clients larger than a given threshold are considered.

A numerical example is provided. Content owner Charlie's Studio usesCDNs A and B for delivering content. At some point, there are 1100streams from ISP-1 in San Francisco, 1200 streams from ISP-2 in LosAngeles, 1500 streams from ISP-3 in New York, 1050 streams from ISP-4 inBoston, 600 streams from ISP-1 in Denver, with exactly half the users ineach group streaming from CDN A and the other half from CDN B. A minimumgroup size of 500 users streaming from a particular (ASN, CDN, Geo)group is used, and hence there are four groups to consider. Arequirement of at least four pairs (K=4) exists. For CDN A, thebuffering ratios for users in San Francisco, Los Angeles, and Boston is0.3, and for users in Denver it is 0.1. For CDN B, the buffering ratioof all user groups is 0.05. The quality threshold to separate good andbad quality is 0.1, and the fraction F required is 75%. This conditionis satisfied, since three out of the four groups have a qualitydifference between CDN A and CDN B as 0.25. Hence, a conclusion is madethat CDN A is having problems streaming to users.

Example: Multiple CDNs with CDN Optimization

Suppose a content owner, such as XYZ, distributes its content viamultiple CDNs (e.g., CDN A, CDN B, and CDN C). Assume that if a clientconnected to CDN A experiences buffering beyond a threshold, it switchesto another CDN (and stays there for at least a threshold interval oftime). Based on the amount of switching observed from a CDN, therelative extent of quality problems the CDNs have can be quantified. Oneway this analysis can be implemented on content distribution monitor 102is as follows.

For each CDNi for a period T, determine the number of clients Ni thatstart with CDNi, and the number of clients Mi that start with CDNi andswitch away from CDNi (i.e. count only the first switch of a client).Compute the switch fraction SWi=Mi/Ni for all CDNs. If(SWi−avg(SW)>SW_Threshold). A conclusion can be made that CDNi hasquality problems in general. Similar analysis can also be performed withrespect to a geographic region or an ASN restriction.

A numerical example is provided. A content owner Charlie's Studios usesCDN A and CDN B to deliver content. Over a 10-minute period, 10,000users watch content starting streaming from CDN A, and in the course ofviewing, 1000 users switch to CDN B based on client-based qualitydetection algorithm. In the same period, 10,000 users watch the samecontent starting streaming from CDN B, and out of these, 2,500 switch toCDN B. The switch fraction for CDN A is 0.1 and that for CDN B is 0.2.Switching threshold SW_Threshold is 0.1, and a conclusion is made thatCDN A has quality problems in general.

Other content distribution problems can also be detected using thetechniques described herein. For example, a problem with a content itemitself (e.g., a particular movie) can be inferred if several clients,using different CDNs and different ISPs, experience quality issues withthe same content. As another example, a problem with ad server 150 canbe inferred if several clients report excessively long advertisementload times or timeouts.

A numerical example is provided. A content owner publishes 100 clipsdaily using two CDNs: CDN A and CDN B. One of the clips, clip X, has100% join failures on both CDNs. Using this information, an inferencecan be made that there is a problem in publishing clip X. Now assumethat there is 100% join failure for clients that join CDN A, but <2% ofthe users have problems with CDN B. Then an inference can be made thatCDN A's publishing path has problems for that clip (but CDN B's doesnot).

Other content distribution problems can also be detected using thetechniques described herein. For example, a problem with a content itemitself (e.g., a particular movie) can be inferred if several clients,using different CDNs and different ISPs, experience quality issues withthe same content. As another example, a problem with ad server 150 canbe inferred if several clients report excessively long advertisementload times or timeouts.

In various embodiments, when problems are detected by contentdistribution monitor 102, corrective actions are automatically taken,either by a control module included in content distribution monitor 102,or by a separate control system, configured to work in conjunction withcontent distribution monitor 102.

FIG. 7 illustrates an example of a process for correcting a problem in acontent distribution. In some embodiments the process shown in FIG. 7 isperformed by content distribution monitor. The process begins at 702when a determination is made that a problem in a content distributionexists. In some embodiments the processing of portion 702 of FIG. 7occurs in accordance with the process shown in FIG. 6. At 704, adetermination is made that at least one setting of a client should beupdated, and at 706, the update is sent to the client.

As one example, suppose content distribution monitor 102 continuouslymonitors the quality of clients' receipt of content. If this qualitydrops under a predefined threshold, the control module (or controlsystem) will try to localize the problem to one or more (CDN, city)pairs, as described above. If successful, the control module instructsclients in each city to use a different CDN. As another example, thecontent distribution monitor can be configured to monitor audiencequality for triplets (CDN, city, ISP), and based on the data inferwhether a quality issue is due to a client's CDN, ISP, or both.

The communication between the control module and the clients can beimplemented in a variety of ways. In one example, the control moduleupdates a centrally maintained configuration file that is periodicallyread by each client. In another example, the control module opens aconnection to any clients implicated in the problem and sends commandsdirectly to those clients. In various embodiments, instead ofspecifically instructing the client to make a change, a list ofalternatives or suggestions is instead provided to the client, and theclient is configured to perform some local decision making. As oneexample, suppose a client is experiencing difficulties obtainingadvertisements from advertisement server 150. If multiple otheradvertising servers exist, rather than content distribution monitor 102determining which advertisement server the client should switch tousing, content distribution monitor 102 sends a list of three suchserves, and the client is tasked with selecting from the list.

If the problem is determined to be CDN A, and CDN A is the only CDN ableto provide the content, in some embodiments the control module instructsthose clients affected by CDN A's problem to reduce the rate at whichthey stream. In addition, new clients joining CDN A can be instructed touse a lower rate than they might otherwise use.

If the problem is determined to be CDN A, and multiple CDNs are able toprovide the content, in some embodiments the control module instructsthose clients affected by CDN A's problem to switch to another CDN. Inaddition, new clients joining CDN A can be instructed to avoid CDN A.

If the problem is determined to be an ASN, in some embodiments thecontrol module instructs the clients connected to that ASN to reduce therate at which they are streaming. In addition, newly clients joiningfrom the ASN can be instructed to use the lower rate.

If the problem is determined to be the client (e.g., the client'sconnection is congested or the CPU is overloaded), in some embodimentsthe control module instructs the client to reduce the rate at which itstreams.

If the problem is determined to be an ad server, in some embodiments thecontrol module instructs all clients to cease fetching or attempting tofetch advertisements from the ad server, and instead to fetch them froma different ad server.

In various embodiments, clients include logic to interpret the controlmodule instructions and information. A client makes the ultimatedecision based on the instructions and information received from controlmodule information and/or its own internal state. The state can includeboth real-time state as well as historical state. Examples of statesinclude, but are not limited to CPU utilizations, rendering rate, andwhether the viewer watches full screen or not.

Selecting a CDN Based on External Input

Suppose a content owner distributes its content via multiple CDNs (CDN Aand CDN B). Suppose also that the content owner's desire is to use CDN Afor clients in the United States and CDN B for clients in Europe. When aclient connects to the content owner's website to view a stream, anentity on the server side determines the location of the client andreturns the CDN that the client should used based on its location. Thistechnique can also be used with respect to more sophisticated policies,such as ones based on content being watched, or user class (e.g.,premium versus regular).

Selecting a CDN Based on External Input of Multiple CDNs

FIG. 8 illustrates an example of an environment in which content isdistributed. In the example shown, a content owner distributes itscontent via multiple CDNs—CDN A (808) and CDN B (810). Suppose thecontent owner's policy is to use the CDN that provides the best qualityfor a client at the current time. When client A (812) connects to thecontent owner's website to view a stream, an entity at the backenddetermines the CDN that is expected to provide the best quality toclient A. The backend entity (814) determines this using performanceinformation from other clients such as clients B (816) and C (818). Thebackend entity sends the list of CDNs to client A in the preferred orderbased on the expected performance for client A. Client A then choosesthe CDN based on this ordered list and local state. In some embodimentsthe local state keeps a black list of CDNs specific to client A based onattempts made by client A to each CDN. One reason for doing this is forwhen a CDN is expected to perform well for a client based on informationknown to the backend entity, but does not perform well in the particularcase for a client. Once client A chooses a CDN, it connects and streamsfrom that CDN.

Selecting Bit Rate Based on External Input and Local State

Suppose a content owner distributes its content using multiple bitrates(e.g., 300 kbps, 700 kbps, 1500 kbps, and 2700 kbps). Suppose also thatthe content owner wants to make all bitrates available to its premiumcustomers in country A, but only make the 300 kbps and 700 kbps bitratesavailable to its regular customers in country A. In country B, thecontent owner wants to implement similar rules, but not provide the 2700kbps bitrate to any customer. For security reasons, the knowledge ofwhether a customer is premium or not is maintained at the contentowner's servers rather than at the client. One example of a premiumcustomer is one who pays more for a higher class of service. When aclient connects to the content owner's servers to stream the video, anentity on the server side determines the set of bitrates available tothe client based on its location and service class and returns this set.The client then selects one of the available bitrates based on localstate on the highest bitrate it can play without experiencing qualityproblems. This state can be maintained at the client based on previousviewings of content on this site or different sites.

Placing an Advertisement on a Page Based on External Input

Suppose a content owner wants to experiment with whether placing anadvertisement on the left side of the page or the right side of the pagehas a better chance of the user clicking on the advertisement. When aclient connects to the content owner's website, an entity on the serverside determines which side the client should place its ad. The clientplaces its ad in the specified location and reports back if the ad wasever clicked.

Treating Inputs from an External Entity as Commands

Suppose a content owner distributes content via multiple contentdistribution networks (CDN A and CDN B). Suppose also that the contentowner wants to tightly control the traffic usage of the two CDNs andwants to change the usage between the CDNs based on quality and pricingin a continuous manner. In an extreme scenario the content owner maywish to turn off a CDN completely and migrate all users to the other CDNwithin a short period of time (e.g. a few minutes). To achieve this, theclient periodically (e.g., every minute) sends a request to the contentowner's website. An entity on the sever side determines for each clientthe CDN it should be connected to based on current policy settings. Onthe response to the first request from a client, the client will connectto the CDN provided. On subsequent responses, if the client is connectedto a different CDN than the one returned, then it will immediatelyswitch to the one returned. Here the client treats the response as acommand to make sure the policies in the back end are enforced.

Treating Inputs from an External Entity as Hints

Suppose a content owner distributes content using multiple contentdistribution networks (CDN A and CDN B) and multiple bitrates (300 kbps,700 kbps, and 1500 kbps). Suppose also that the player used by thecontent owner automatically adjusts the bitrate to the highest bit ratethat the client can stream from the current CDN but does not switchbetween CDNs if streaming is working well. With this setup, it ispossible that the client will get into the scenario where it plays the700 kbps bitrate well on CDN A but cannot play the 1500 kbps on CDN A.Now suppose that the client could play at 1500 kbps at CDN B, but theclient does not know this. In various embodiments, a “hint” from anexternal entity that has knowledge of this possibility (e.g., throughinference based on other clients) can be sent to this client letting theclient know that it may be able to play a higher bit rate on CDN B. Theclient may choose to ignore or take this hint based on local state suchas user settings.

Decision on CDN is Updated Periodically

Suppose a content owner distributes its content via multiple CDNs (CDN Aand CDN B). Suppose also that the content owner's policy is to use theCDN that provides the best quality for a client at the current time.When client A connects to the content owner's website to view a stream,an entity at the backend determines the CDN that is expected to providethe best quality to client A. The backend entity determines this usingperformance information from other clients (in this example clients Band C). The backend entity sends the list of CDNs to client A in thepreferred order based on the expected performance for client A. Client Athen chooses a CDN based on this ordered list and local state. Onceclient A chooses a CDN, it connects to and streams from that CDN.Network and CDN performance change over time and the CDN selectiondecision is updated periodically accordingly. Client A periodically(e.g., once every one minute) requests a new list of CDNs from thebackend entity. If the backend entity determines that the CDN currentlybeing used by client A is no longer best suited for client A, it willreturn a list with a different CDN as the most preferred CDN. Whenclient A receives this new list, it makes a decision using the new listand local state on whether to stay on the current CDN or switch to thenew one. If it decides to switch, it will immediately switch to the newmost preferred CDN.

Client Decision when Communication with Backend Entity is Lost

Suppose a content owner distributes its content via multiple CDNs (CDN Aand CDN B). Suppose also that the content owner's policy is to use theCDN that provides the best quality for a client at the current time.When client A connects to the content owner's website to view a stream,an entity at the backend determines the CDN that is expected to providethe best quality to client A. The backend entity determines this usingperformance information from other clients (in this example clients Band C). The backend entity sends the list of CDNs to client A in thepreferred order based on the expected performance for client A. Client Athen chooses the CDN based on this ordered list and local state. Onceclient A chooses a CDN, it connects to and streams from that CDN. Duringnormal operation, if client A detects a quality problem with the CDN itis streaming from, it notifies the backend entity to get a differentCDN. However, in the event it loses connectivity with the backendentity, the client uses the next CDN in the most recent list provided bythe backend entity to select the next CDN to try.

The control module can use the information collected from the clients todetermine the correlation between the viewer engagement and quality.Alternatively, an operator can use the historical or real-timeinformation to determine this correlation.

Quantitative Relationship Between Video Quality and Viewer Engagement

Viewer sessions can be classified based on a set of (N) contentattributes such as: (a) content format; (b) content type; (c) contentgenre; (d) content length; and (e) content publish time.

Engagement difference can be computed for viewer sessions viewing“similar” content. One definition of content “similarity” is equality ofa majority of content attributes where (a) all (N) attributes are equal;(b) only (N−1) attributes may be equal; (c) only (N−2) attributes may beequal; etc.

By computing and comparing engagement differences for content withsimilar attributes (where a majority of attributes are similar), theimpact of content itself on engagement can be reduced, thus highlightingthe impact of video quality on viewer experience.

Example 1

Consider a live sporting event of duration D (e.g., D=2 hours). Let thenumber of viewers joining to watch the event in the first minute beN=10,000. Let the average video quality and corresponding viewerengagement of various viewers be as follows:

TABLE 1 Viewer Number Engagement of (Viewing Category Viewers ViewersVideo Quality Duration) C1 V1 . . . N1 = Join Time = 1 sec, D1 = 1 hr 10V3000 3000 Buffering Ratio = 0, mins Rendering Quality = 100% C2 V3001 .. . N2 = Join Time = 4 secs, D2 = 1 hr V7000 4000 Buffering Ratio = 3%,Rendering Quality = 95% C3 V7001 . . . N3 = Join Time = 4 secs, D3 = 40mins V10000 3000 Buffering Ratio = 6%, Rendering Quality = 90%

As shown in TABLE 1, viewers have been classified by video quality intothree categories. Viewers with poorer video quality watched the videofor fewer minutes. The net loss of viewer engagement for viewers withpoor video quality can be computed as: [N2× [D1−D2]]+[N3×[D3−D1]]=4000×10+3000×30=13,000 minutes.

Example 2

Consider a website where the viewer can view multiple videos on demand.After viewing a video, the viewer is presented the same interface tooptionally continue watching more videos. Let there be a set of K videoswith similar attributes (genre=“action”, format=“episode”, duration=“30mins”). Let the number of viewers who have watched at least one of thesevideos in a given month be N=10,000. Let the average number of videoswatched by viewers with varying video quality be as follows:

TABLE 2 Viewer Engagement (Average Number Number of of Videos WatchedCategory Viewers Viewers Video Quality in a Month) C1 V1 . . . N1 = JoinTime = 1 sec, D1= 3.5 V3000 3000 Buffering Ratio = 0, Rendering Quality= 100% C2 V3001 . . . N2 = Join Time = 4 secs, D2 = 3   V7000 4000Buffering Ratio = 3%, Rendering Quality = 95% C3 V7001 . . . N3 = JoinTime = 4 secs, D3 = 2   V10000 3000 Buffering Ratio = 6%, RenderingQuality = 90%As shown in TABLE 2, viewers have been classified by video quality intothree categories. Viewers with poorer video quality watched a fewernumber of videos. The net loss of viewer engagement for viewers withpoor video quality can be computed as [N2×[D1−D2]]+[N3× [D3−D1]]×(duration of episode)=(4000×0.5+3000×1.5)×30=195,000 minutes.

Visualization Tools and Manual Diagnosis of Problems

In addition to automatically detecting content distribution problems,content distribution monitor 102 is configured to make availablevisualization tools to human operators. Using the tools, such operatorscan quickly identify problems. The interface creates visualizations forthem based on quality metrics such as the percentage of viewers in abuffering state, the amount of time needed to start playing a videostream as perceived by the viewers, etc. Other users, such as businessand marketing executives can also use the tools to examine data trendsin general, and without needing a specific purpose of problem detection.The interface creates visualizations for them based on metrics that arecorrelated with revenue and brand. Examples include the concurrentnumber of viewers, the percentage of viewers dropping because of qualityproblems, the number of advertising clicks, etc. Interactive controlsare provided to allow both operators and other users to view virtuallyany slice of the multidimensional data captured by content distributionmonitor 102.

FIGS. 9-13 illustrate embodiments of an interface through which contentdistribution monitoring data is exposed. In various embodiments, theinterfaces are provided by content distribution monitor 102, such asthrough gateway layer 526.

The interface shown in FIG. 9 includes four plots. The “Audience Size”plot (upper-left) shows the total number viewers versus time. The“Resource Utilization” plot (upper-right) shows the total bandwidthdelivered by each CDN. The “Audience Quality” plot (bottom-left) showsthe aggregate quality experienced by all viewers. The “AudienceGeolocation” plot (bottom-right) shows the location of the viewers on aworld map. Each plot shows data in realtime, and changes are reflectedthrough frequent refreshes of the interface.

Suppose an operator sees audience quality dropping, as shown in FIG. 10.This drop in quality correlates with a drop in the total number ofviewers, and a drop of the bandwidth delivered by CDN A. Based on thisdata, the operator hypothesizes that CDN A is likely part of theproblem.

The operator can verify this hypothesis by creating two new plots, asshown in FIG. 11. The upper-right plot shows the audience quality rankedby CDNs, while the bottom-left plot shows the audience quality ranked bycity. Since, in both of these plots, CDN A and cities in the Seattleregion are the only ones to perform poorly, the operator furtherhypothesizes that the problem is localized to the viewers in the Seattleregion streaming from CDN A.

To verify this new hypothesis, the operator adds a new plot showing thequality of viewers in Seattle ranked by CDN, as shown in thebottom-right plot in FIG. 12. This plot show that, indeed, only theSeattle viewers who stream from CDN A see quality issues; viewers fromSeattle streaming from other CDNs see no quality degradation. As aresult, the operator concludes that the hypothesis was correct, i.e.,that the problem is localized to the viewers in the Seattle regionstreaming from CDN A.

The operator can take actions to improve the viewer quality in theSeattle region. Examples of such actions include calling CDN A to fixthe problem, or/and pushing new configuration files to the players. Suchconfiguration file can, e.g., specify that if the player is in theSeattle region, it should use a CDN other than CDN A.

In addition to diagnosing widespread problems, operators can also usethe visualization tools provided by content distribution monitor 102 todiagnose quality issues experienced by particular clients.

Suppose a user who reports quality issues to a customer serviceorganization in charge with overseeing the delivery of the content. Theuser provides the service operator with the IP address or other uniqueidentifier (e.g., HTTP cookie, user login) to help the operator identifythe user's end-host.

The operator uses this identifier to see in real-time a summary of theinformation associated with the user's client. FIG. 13 shows an examplein which the quality of the client with the IP address 123.456.7.8 isonly 10% (upper-left plot). In addition, the interface indicates thatthe client is located in Seattle and streams data from CDN A.

To determine the root cause the problem the operator uses the interfaceto show (a) the audience quality of viewers using CDN A (upper-rightplot), and (b) the audience quality of all viewers in Seattle ranked byCDN (bottom-left plot).

Based on the second plot that shows that only the viewers in Seattlewhich stream from CDN A experience quality issues, the operatorconcludes that likely the problem experienced by the client is due toCDN A. To fix this problem, the operator configures the client remotely(e.g., via a configuration file) to use another CDN.

Additional Embodiments Multivariate Testing

The techniques described herein can be used for a variety of purposes inaddition to detecting and remedying content distribution problems. Asone example, content owners (or other appropriate parties) can performmultivariate testing and, based on the results, make adjustment tocontent distribution settings.

FIGS. 14A and 14B are graphs illustrating the impact of distributionquality on a content item. Specifically, FIG. 14A indicates that, for afirst content item (e.g., a feature length film), clients whichexperience a good experience tend to watch the item approximately twiceas long as those with a poor experience. FIG. 14B indicates that, for asecond content item (e.g., a breaking news report), the length of time aparticular client streams the content is largely unimpacted by thequality of the experience.

FIG. 15 illustrates an example of a process for improving a contentplayer engagement. The process begins at 1402 when an engagement of afirst content player with respect to a first content item beingdownloaded by the first client is measured. As one example, the amountof time a particular client, such as client 176, views a particularfeature length film is determined. At 1404, heartbeat information isreceived from the client. In some embodiments the processing of 1402 and1404 is combined, as applicable. At 1406, a determination is made as tothe quantitative relationship between the engagement and the performanceinformation. In various embodiments, the processing of 1402 and 1404 isalso performed with respect to other clients that are also accessing thesame content, and the resulting information is used at 1406. At 1408, asetting of a second player with respect to the same content is adjusted.As one example, if a determination is made that the content can beprovided, at reduced quality, without incurring a significant reductionin engagement, the content owner may instruct content distributionmonitor 102 to send instructions to other clients receiving the contentto obtain the content at a lower bitrate. Similarly, if a determinationis made that changes in quality significantly change engagement,instructions to devote more bandwidth to clients experiencing poorperformance might be provided to content distribution monitor 102.

Enforcing Policies

In various embodiments, content owners and other appropriate parties aregranted access to a policy engine that allows them to configure andrevise a list of policies. Policies provided by the customer are basedon the following pattern:

1. matching rule A: action A

2. matching rule B: action B

3. matching rule C: action C

The matching rule determines the subset of the viewers on which thepolicy is to be applied. In addition, multiple matching rules can beprovided for certain policies according to a specified priority order.When the policy engine needs to decide an action for a viewer, ititerates through the ordered list and based on the first match selectsthe appropriate action.

The matching rules are composed of predicates on several dimensions thatidentify the viewers. Example dimensions include:

1. Location of the viewer (e.g., Country, State, City, Zip code).

2. The ISP and AS through which the viewer is connecting to theInternet.

3. Technographics of the viewer (e.g., browser, OS, network speed)

4. Content being watched by the viewer (e.g., name of video, play listname, etc.)

Example actions include:

1. Maintain uninterrupted viewing by switching between an ordered listof resources (e.g., Data centers, CDNS) via which the viewer downloadsthe video. This provides high availability of the connection of theviewer to the content.

2. Maintain high quality viewing by switching in accordance with anordered list of resources via which the viewer downloads the video. Thisprovides a glitch free viewing experience (e.g., minimizes the impact ofoverloaded delivery servers, or network path congestion).

3. Optimize video resolution by switching the bitrate of the video. Thisallows the download bandwidth of the viewer to provide the highestresolution possible.

4. Optimize video viewing quality given limited bandwidth available tothe user by using a lower bitrate video. This provides consistentdelivery of content even if the bandwidth available to the user is low.

5. Minimize cost of delivery by switching the viewer to lower deliverycost resource if the quality of download offered by the lower costresource is consistent.

6. Maintain load balance between a list of available resources. The loadbalancing is done based on priority weights associated with eachresource in the list.

7. Enforce a usage limit on a resource by forcing viewers to switch awayafter a bandwidth or downloaded byte threshold is met.

As mentioned above, in various embodiments, content distribution monitor102 includes a control module or works in conjunction with a separatecontrol system. The control system is configured to enforce policiesspecified via the policy engine. As one example, in some embodimentswhenever an initial client request is made for content, the controlsystem determines an appropriate content source for the client based onthe policies and also based on telemetry information made available viacontent distribution monitor 102.

Although the foregoing embodiments have been described in some detailfor purposes of clarity of understanding, the invention is not limitedto the details provided. There are many alternative ways of implementingthe invention. The disclosed embodiments are illustrative and notrestrictive.

What is claimed is:
 1. A system, including: one or more processorsconfigured to: measure an engagement of a first content player withrespect to a content item being downloaded by a first remote client;obtain performance information associated with the first content player;based at least in part on the measured engagement of the first contentplayer with the content item and the performance information associatedwith the first content player, determine, for the content item, aquantitative relationship between content quality and content playerengagement; and based at least in part on the quantitative relationshipdetermined, for the content item, between content quality and contentplayer engagement, adjust, for a second remote client receiving the samecontent item, a setting associated with at least one of a download and aplaying of the same content item by the second remote client, whereinthe adjusting is performed at least in part by sending an instruction tothe second remote client; and a memory coupled to the one or moreprocessors and configured to provide the one or more processors withinstructions.
 2. The system of claim 1 wherein the measured engagementcomprises a length of time that the first content player has beenplaying the content item.
 3. The system of claim 1 wherein the measuredengagement comprises a length of time that the first content player hasbeen playing a plurality of content items.
 4. The system of claim 1wherein the measured engagement comprises a length of time that thefirst remote client has been connected to a content distributionnetwork.
 5. The system of claim 1 wherein the measured engagementcomprises an amount of data associated with a download of the contentitem.
 6. The system of claim 1 wherein the measured engagement comprisesa number of content items viewed.
 7. The system of claim 1 wherein themeasured engagement comprises a display mode of a video screen.
 8. Thesystem of claim 1 wherein the measured engagement comprises a viewerinteraction with at least one player control.
 9. The system of claim 1wherein the measured engagement comprises a viewer interaction with anadvertisement.
 10. The system of claim 1 wherein the obtainedperformance information comprises information associated with a joinfailure.
 11. The system of claim 1 wherein the obtained performanceinformation comprises information associated with a join time.
 12. Thesystem of claim 1 wherein the obtained performance information comprisesinformation associated with a number of buffering events.
 13. The systemof claim 1 wherein the obtained performance information comprisesinformation associated with a duration of buffering events.
 14. Thesystem of claim 1 wherein the obtained performance information comprisesinformation associated with a frequency of buffering events.
 15. Thesystem of claim 1 wherein the obtained performance information comprisesinformation associated with a frequency of frames per second of videorendered.
 16. The system of claim 1 wherein the obtained performanceinformation comprises information associated with a rendering quality.17. The system of claim 1 wherein the obtained performance informationcomprises information associated with a video bit rate.
 18. The systemof claim 1 wherein a bitrate is adjusted.
 19. The system of claim 1wherein a source of the content item is adjusted.
 20. The system ofclaim 19 wherein the source is a content delivery network.
 21. Thesystem of claim 1 wherein a buffer size is adjusted.
 22. The system ofclaim 1 wherein the one or more processors are further configured toadjust a setting associated with a download of the content item by thefirst remote client.
 23. The system of claim 1 wherein the obtainedperformance information is a first information and wherein the one ormore processors are configured to obtain additional performanceinformation associated with a plurality of respective additional contentplayers.
 24. The system of claim 1 wherein the one or more processorsare further configured to measure an engagement of a plurality ofcontent players with respect to the same content item.
 25. The system ofclaim 1 wherein the one or more processors are further configured tomeasure an engagement of a third content player with respect to anothercontent item.
 26. The system of claim 25 wherein the first and thirdcontent player are different.
 27. A method, including: measuring anengagement of a first content player with respect to a content itembeing downloaded by a first remote client; obtaining performanceinformation associated with the first content player; based at least inpart on the measured engagement of the first content player with thecontent item and the performance information associated with the firstcontent player, determining, for the content item and using one or moreprocessors, a quantitative relationship between content quality andcontent player engagement; and based at least in part on thequantitative relationship determined, for the content item, betweencontent quality and content player engagement, adjust, for a secondremote client receiving the same content item, a setting associated withat least one of a download and a playing of the same content item by thesecond remote client, wherein the adjusting is performed at least inpart by sending an instruction to the second remote client.
 28. Acomputer program product embodied in a non-transitory computer readablestorage medium and comprising computer instructions for: measuring anengagement of a first content player with respect to a content itembeing downloaded by a first remote client; obtaining performanceinformation associated with the first content player; based at least inpart on the measured engagement of the first content player with thecontent item and the performance information associated with the firstcontent player, determining, for the content item, a quantitativerelationship between content quality and content player engagement; andbased at least in part on the quantitative relationship determined, forthe content item, between content quality and content player engagement,adjust, for a second remote client receiving the same content item, asetting associated with at least one of a download and a playing of thesame content item by the second remote client, wherein the adjusting isperformed at least in part by sending an instruction to the secondremote client.